Aerial vehicles with machine vision

ABSTRACT

An aerial vehicle is provided. The aerial vehicle can include a plurality of sensors mounted thereon, an avionics system configured to operate at least a portion of the aerial vehicle, and a machine vision controller in operative communication with the avionics system and the plurality of sensors. The machine vision controller is configured to perform a method. The method includes obtaining sensor data from at least one sensor of the plurality of sensors, determining performance data from the avionic system or an additional sensor of the plurality of sensors, processing the sensor data based on the performance data to compensate for movement of the unmanned aerial vehicle, identifying at least one geographic indicator based on processing the sensor data, and determining a geographic location of the aerial vehicle based on the at least one geographic indicator.

FIELD

The present subject matter relates generally to aerial vehicles and, inparticular, to aerial vehicles and other aircraft with machine vision.

BACKGROUND

Generally, it is useful to know a geographic location of an aircraftrelative to a desired landing position or area, such as a runway,landing pad, or other suitable landing area. During flight, the positionof the aircraft relative to the desired landing area aids insuccessfully landing the aircraft. Furthermore, understanding anapproach vector during a landing attempt may further aid in safelylanding the aircraft.

While aircraft generally have a plurality of instrumentation availableto aid pilots and operators of the aircraft to safely maneuver theaircraft during flight, there may be limited instrumentation to aid inlanding in other scenarios. There may be limited aid frominstrumentation when a position is unknown, there is decreasedvisibility from a control position of the aircraft, and/or other issuesin determining a position of an aircraft are apparent. As an example,limited visibility may result in a pilot misidentifying a runway orlanding area. Furthermore, limited visibility may result in an airportrunway obstruction going unnoticed by a pilot or support crew. Moreover,other conditions may result in misidentification of a landing area orlanding in dangerous conditions.

Current systems require complicated radar systems, global positioningsystems (GPS), and communication methodologies in an attempt to avoidthese and other potential issues. Many of these systems are dependent onresources external to the aircraft and communication with the aircraft,making them subject to loss of utility if communication with a pilot islost. Additionally, many GPS-based systems will not provide any helpfulinformation with regards to obstructions, unsafe landing conditions, ormisidentification of a runway if a general position (based on GPS) iscorrect.

BRIEF DESCRIPTION

Aspects and advantages of the disclosed technology will be set forth inpart in the following description, or may be obvious from thedescription, or may be learned through practice of the disclosure.

According to one example embodiment, an aerial vehicle is provided. Theaerial vehicle can include a plurality of sensors mounted thereon, anavionics system configured to operate at least a portion of the aerialvehicle, and a machine vision controller in operative communication withthe avionics system and the plurality of sensors. The machine visioncontroller can be configured to perform a method. The method can includeobtaining sensor data from at least one sensor of the plurality ofsensors. The sensor data is associated with a proximal location of theaerial vehicle and the proximal location is within sensor range of theaerial vehicle. The method can also include determining performance dataassociated with the aerial vehicle, processing the sensor data based onthe performance data to compensate for movement of the unmanned aerialvehicle, identifying at least one geographic indicator based onprocessing the sensor data, and determining a geographic location of theaerial vehicle based on the at least one geographic indicator.

According to another example embodiment, an unmanned aerial vehicle isprovided. The unmanned aerial vehicle can include a plurality of sensorsmounted thereon, an avionics system configured to operate at least aportion of the unmanned aerial vehicle, and a machine vision controllerin operative communication with the avionics system and the plurality ofsensors. The machine vision controller can be configured to perform amethod. The method can include obtaining sensor data from at least onesensor of the plurality of sensors. The sensor data is associated with aproximal location of the unmanned aerial vehicle and the proximallocation is within sensor range of the unmanned aerial vehicle. Themethod can also include identifying at least one geographic indicatorbased on processing the sensor data, determining whether the at leastone geographic indicator is associated with a flight plan of theunmanned aerial vehicle, and performing an automatic maneuver of theunmanned aerial vehicle based on the determining.

According to another example embodiment, a method of locating an aerialvehicle relative to a proximal location is provided. The method caninclude obtaining sensor data from one or more optical sensors mountedon the aerial vehicle. The sensor data is associated with the proximallocation and the proximal location is within sensor range of the aerialvehicle. The method can also include determining performance data fromone or more additional sensors mounted on the aerial vehicle, processingthe sensor data based on the performance data to compensate for movementof the unmanned aerial vehicle, identifying at least one geographicindicator based on processing the sensor data, and determining ageographic location of the aerial vehicle based on the at least onegeographic indicator.

These and other features, aspects and advantages of the disclosedtechnology will become better understood with reference to the followingdescription and appended claims. The accompanying drawings, which areincorporated in and constitute a part of this specification, illustrateembodiments of the disclosed technology and, together with thedescription, serve to explain the principles of the disclosedtechnology.

BRIEF DESCRIPTION OF THE DRAWINGS

A full and enabling disclosure of the present disclosure, including thebest mode thereof, directed to one of ordinary skill in the art, is setforth in the specification, which makes reference to the appendedfigures, in which:

FIG. 1 is a schematic illustration of an example runway or landing areafor an aerial vehicle.

FIG. 2A is a schematic illustration of an aerial vehicle approaching aportion of the landing area of FIG. 1.

FIG. 2B is a schematic illustration of example sensor data obtained bythe aerial vehicle of FIG. 2A.

FIG. 3A is a schematic illustration of an additional aerial vehicleapproaching a portion of the landing area of FIG. 1.

FIG. 3B is a schematic illustration of example sensor data obtained bythe aerial vehicle of FIG. 3A.

FIG. 4 is a diagram of an example aerial vehicle, according to exampleembodiments of the present disclosure.

FIG. 5 is a flow diagram of a method of machine vision processing ofsensor data of an aerial vehicle, according to example embodiments ofthe present disclosure.

FIG. 6 is a flow diagram of an additional method of machine visionprocessing of sensor data of an aerial vehicle, according to exampleembodiments of the present disclosure.

FIG. 7 is a block diagram of an example computing system that can beused to implement methods and systems according to example embodimentsof the present disclosure.

DETAILED DESCRIPTION

Reference now will be made in detail to embodiments of the disclosedtechnology, one or more examples of which are illustrated in thedrawings. Each example is provided by way of explanation of thedisclosed technology, not limitation of the disclosed technology. Infact, it will be apparent to those skilled in the art that variousmodifications and variations can be made in the present disclosurewithout departing from the scope or spirit of the claims. For instance,features illustrated or described as part of example embodiments can beused with another embodiment to yield a still further embodiment. Thus,it is intended that the present disclosure covers such modifications andvariations as come within the scope of the appended claims and theirequivalents.

As used in the specification and the appended claims, the singular forms“a,” “an,” and “the” include plural referents unless the context clearlydictates otherwise. The use of the term “about” in conjunction with anumerical value refers to within 25% of the stated amount.

In example aspects, a machine vision controller for an aerial vehicle oraircraft is provided. In some embodiments, the machine vision controllermay execute computer-readable program instructions for: processingsensor data, identifying geographic identifiers in sensor data,comparing obtained sensor data to desired sensor data, identifyinggeographic location from sensor data, automatically maneuvering theaerial vehicle responsive to sensor data and other data, determining ifan approach vector is safe for landing, reducing risk associated withinstrumentation-only landings, and/or other program instructions. Themachine vision controller may be integrated within an aerial vehicle ormay be a remote controller configured to transmit calculated data to theaerial vehicle. Additionally, the machine vision controller may also beconfigured to provide warnings, both audial and visual, to operators ofthe aerial vehicle, to aid in a plurality of aircraft maneuvers.Hereinafter, a detailed description of several example embodiments ofaerial vehicles and machine vision controllers are provided in detail.

Embodiments of the disclosed technology provide a number of technicalbenefits and advantages, particularly in the area of aircraft safety. Asone example, the disclosed technology provides for safer landings byensuring an aerial vehicle is approaching an appropriate runway orairport. The disclosed technology can also aid in automatic avoidancemaneuvers through identifying geographic indicators. By identifyinggeographic indicators and matching geographic indicators to a map ormodel of an expected proximal area, the disclosed systems can avoidinitiating a landing operation if an area is unexpected or if obstaclesexist.

Embodiments of the disclosed technology additionally provide a number oftechnical benefits and advantages in the area of computing technology.For example, the disclosed system can obtain sensor data associated withperformance data of an aerial vehicle and automatically determinewhether performance of an aircraft is affecting a landing approach. Acomputing system implemented in accordance with the disclosed technologycan therefore avoid costly landing maneuvers in adverse conditions orimplement safer landing procedures through assuring an actual approachvector is correct for a particular aerial vehicle.

FIG. 1 is a schematic illustration of an example landing area 2 for anaerial vehicle. Generally, the landing area 2 can be aligned with arunway 4. The runway 4 may include several identifying markers paintedthereon, including several indicia. For example, the runway 4 caninclude runway threshold markings 6, runway designation markings 8,runway aiming point markings 10, runway touchdown zone markings 12,runway centerline markings 14, runway side stripe markings 16, runwaylighting 18, taxiway markings including taxiway centerline 20, taxiwayedge marking 22, taxiway lighting 24, holding position markings 26,holding position sign 28, and holding position sign 30. The identifyingmarkers and indicia may be processed by the machine vision controllersdescribed herein, to aid in both taxying an aerial vehicle prior totakeoff and safely landing an aerial vehicle during/subsequent to alanding approach.

In some circumstances, for example during relatively poor weather orwith limited visibility, an aerial vehicle may not have full visibilityof some or all of the identifying markers outlined above. In thesecircumstances, it may be beneficial for an aerial vehicle to obtainsensor data, such as optical data, video data, photograph data, radardata, and/or Light Detection and Ranging (LIDAR) data. The sensor datacan subsequently be processed to determine if an aerial vehicle cansafely land or if an avoidance maneuver is appropriate. Additionally,under some circumstances, the sensor data can be compared to a map ormodel of a proximal area near the aircraft. Upon comparing, geographicindicators including buildings, landmarks, topographical features,runways, runway markings, and other geographic indicators can beidentified to determine that the aerial vehicle is operating orattempting to land in a safe or expected area. Hereinafter, scenariosinvolving an aerial vehicle approaching the runway 4 are described indetail with reference to FIG. 2A, FIG. 2B, FIG. 3A, and FIG. 3B.

FIG. 2A is a schematic illustration of an aerial vehicle 200 approachinga portion 215 of the landing area of FIG. 1. For example, taxiway 40 andtaxiway 42 with geographic position markings 44 including a directionsign 46 and a location sign 48 are presented on the portion 215.Furthermore, the aerial vehicle 200 is approaching the portion 215 atapproach vector 202.

During the approach, the aerial vehicle 200 may obtain a variety ofsensor data. FIG. 2B is a schematic illustration of example sensor data210 obtained by the aerial vehicle 200. As illustrated, the directapproach vector 202 allows the sensor data 210 to appear relativelyundistorted. In this example, a machine vision controller associatedwith the aerial vehicle 200 can process the sensor data 210 to determinethat a direction indicator 46 and position markings 44 are appropriatelylocated relative to the aerial vehicle 200. In some implementations, themachine vision controller may generate one or more displays for a pilotor other operator providing the direction indicator 46, positionmarkings 44, and/or a representation of any other geographic indicator.

The machine vision controller may utilize various image processingand/or machine vision processing techniques to identify objects usingimage data. Physics-based modeling and/or machine learned models may beused for object detection. Objects may be detected and classified usingvarious image and/or machine vision processing techniques.

Accordingly, the aerial vehicle 200 can safely continue its approach andcomplete landing maneuvers. However, if the aerial vehicle 200 andassociated machine vision controller determine that the approach vector202 is inappropriate or misaligned, the machine vision controllerassociated with the aerial vehicle 200 can provide warnings,indications, and other data to a pilot or operator to ensure the pilotis aware of the situational position and approach of the aerial vehicle200.

Under some circumstances, the warnings may be used to determine that alanding cannot continue due to obstructions, misaligned vectors,incorrect geographical areas, or other misinformation. Accordingly, theaerial vehicle 200 may be equipped to efficiently correct themisinformation, alter a flight plan, and/or avoid a landing to ensurethe aerial vehicle 200 lands at a correct geographic location, safely.Under other circumstances, typical machine vision processing of thesensor data 210 may be inappropriate due to a variety of issues, such asweather or approach vectors differing from the thrust vector of anaerial vehicle.

In some implementations, aircraft performance data may be used toselectively trigger alerts or displays that are generated based on thesensor data such as image data. By way of example, the machine visioncontroller may obtain altitude and/or aircraft configuration data fromthe performance data and use this information to selectively generategeographic based alerts. By way of example, the machine visioncontroller may determine from the performance data whether the aircraftis below a certain altitude and/or whether the aircraft landing gear isdown. If the performance data indicates that the aircraft is attemptinga landing, displays may be generated based on the geographic indicators.For example, runway markers and other identifiers may be displayed ifthe performance data indicates that the aircraft is landing. If,however, the performance data indicates that the aircraft is notattempting a landing, alerts or displays that would otherwise begenerated in response to geographic indicators may not be displayed.

FIG. 3A is a schematic illustration of aerial vehicle 200 approachingthe portion 215 of the landing area of FIG. 1. As illustrated, theaerial vehicle 200 has a thrust vector 302 in an attempt to correct forcrosswind 306 that is flowing perpendicular to the portion 215 of thelanding area of FIG. 1. Thus, while the thrust vector 302 is generallydirected away from the landing area 215, the aerial vehicle 200 isactually on an approach vector 304. Due to the direction of the aerialvehicle being out of perfect alignment with the landing area 215, sensordata obtained by the aerial vehicle 200 may be distorted or difficult toprocess.

As an example, FIG. 3B is a schematic illustration of example sensordata 310 obtained by the aerial vehicle 200 of FIG. 3A on thrust vector302. As shown, due to the thrust vector 302, and therefore the axialalignment of the aerial vehicle 200 not aligning with the portion 215 ofthe landing area, the sensor data 310 appears to point towards adifferent landing area as compared to sensor data 210. In this scenario,a machine vision controller associated with the aerial vehicle 200 canprocess the sensor data 310 to compensate for movement and/orperformance data of the aerial vehicle 200. For example, the machinevision controller can determine a crosswind value 306 and performancedata including the thrust vector 302. The machine vision controller canalso estimate the approach vector 304 based on the crosswind value 306and the performance data. Thereafter, the machine vision controller cantranslate the sensor data 310 into 320 based on the approach vector 304.Therefore, the machine vision controller can correct the apparentmisalignment or distortion based on the thrust vector 302, approachvector 304, and crosswind 306, resulting in sensor data 320. Sensor data320 is sufficiently similar to sensor data 210 and allows for successfulidentification of the portion 215, including all visual markersillustrated thereon. Additionally, under some circumstances, the sensordata 320 can be compared to a map or model of a proximal area near theaerial vehicle 200. The map or model can include a two-dimensionalmapping of information with associated height, depth, or otherdimensional information, for example, of surrounding buildings andlandmarks. Upon comparing, geographic indicators including buildings,landmarks, topographical features, runways, runway markings, and othergeographic indicators can be identified to determine that the aerialvehicle 200 is operating or attempting to land in a safe or expectedarea.

As described above, aerial vehicle 200 may include a machine visioncontroller configured to process sensor data, identify geographicidentifiers in sensor data, compare obtained sensor data to desiredsensor data, identify geographic location from sensor data,automatically maneuver the aerial vehicle responsive to sensor data andother data, determine if an approach vector is safe for landing, and/orreduce risk associated with instrumentation-only landings. Furthermore,under some circumstances, the machine vision controller can predict aflight path of the aerial vehicle 200 based on processing the sensordata. Flight path prediction may include an estimated or predictedflight path based on any of a thrust vector, crosswind, axial alignmentbased on a ground plane reference, instrumentation values, and otheraerial vehicle performance data. The machine vision controller may beintegrated within the aerial vehicle 200 or may be a remote controllerconfigured to transmit calculated data to the aerial vehicle. Themachine vision controller may also be configured to provide warnings,both audial and visual, to operators of the aerial vehicle, to aid in aplurality of aircraft maneuvers. Hereinafter, a detailed description ofan example aerial vehicle 200 having an integrated machine visioncontroller 402 is provided.

FIG. 4 is a diagram of an example aerial vehicle 200, according toexample embodiments of the present disclosure. As illustrated, theaerial vehicle 200 can include a machine vision controller 402configured for sensor data processing as described herein. The machinevision controller 402 can be a standalone controller, customizedcontroller, or can be a generic computer controller configured toprocess sensor data as a machine vision processor.

The controller 402 can be configured to obtain sensor data from a firstsensor array 404. The sensor array 404 may be an externally mountedsensor array or an internal mounted sensor array. The sensor array 402can include one or more of optical sensors, camera sensors, acousticsensors, radar sensors, laser sensors, and/or LIDAR sensors. Opticalsensors can include visible light sensors, infrared sensors, and otheroptical sensors tuned to more or fewer frequencies. For example, opticalsensors that can sense and encode infrared frequencies may have betterperformance characteristics in low-light and inclement weatherconditions as compared to typical human sight. Other appropriate sensorsmay also be included in the sensor array 404, according to any desiredimplementation of the aerial vehicle 200.

The controller 402 can also be configured to obtain sensor data from asecond sensor array 406. The sensor array 406 may be an externallymounted sensor array or an internal mounted sensor array. The sensorarray 406 can include one or more of optical sensors, camera sensors,acoustic sensors, radar sensors, laser sensors, and/or LIDAR sensors.Optical sensors can include visible light sensors, infrared sensors, andother optical sensors tuned to more or fewer frequencies. Otherappropriate sensors may also be included in the sensor array 406,according to any desired implementation of the aerial vehicle 200.

The controller 402 may be configured to communicate with externalprocessors, controllers, and/or ground equipment and other aircraftthrough interface 408. The interface 408 may be a standardizedcommunications interface configured to send and receive data via antennaarray 410. It is noted that although particularly illustrated as anexternally mounted antenna array on the upper surface of the aerialvehicle 200, that any form of antenna array may be applicable.

The controller 402 may also be configured to communicate with avionicssystem 412 of the aerial vehicle 200 over communications bus 450. Forexample, the controller 402 may provide appropriate information toavionics system 412 such that landing gear 422 is controlled up/downbased on machine vision processing. Furthermore, the controller 402 mayprovide appropriate information to avionics system 412 such that thrustfrom engines 424 is controlled based on machine vision processing.Moreover, the controller 402 may provide appropriate information toavionics system 412 such that control surfaces 426 are adjusted forautomatic maneuvers such as landing denials, obstacle avoidance, safetymaneuvers, and other movement of the aerial vehicle 200.

It should be readily understood that the aerial vehicle 200 may includemore or fewer control components than those particularly illustrated.Furthermore, the aerial vehicle 200 may include several components,aspects, and necessary structures not particularly illustrated hereinfor the sake of brevity, clarity, and concise disclosure of the exampleembodiments described herein. Hereinafter, operational details of theaerial vehicle 200 are presented with reference to FIG. 5 and FIG. 6.

The controller and avionics system may generally include one or moreprocessor(s) and associated memory configured to perform a variety ofcomputer-implemented functions, such as various methods, steps,calculations and the like disclosed herein. In some examples, thecontroller and/or avionics system may be programmable logic devices,such as a Field Programmable Gate Array (FPGA), however they may beimplemented using any suitable hardware and/or software.

The term processor may generally refer to integrated circuits, and mayalso refer to a controller, microcontroller, a microcomputer, aprogrammable logic controller (PLC), an application specific integratedcircuit (ASIC), a Field Programmable Gate Array (FPGA), and otherprogrammable circuits. Additionally, the memory described herein maygenerally include memory element(s) including, but not limited to,computer readable medium (e.g., random access memory (RAM)), computerreadable non-volatile medium (e.g., flash memory), a compact disc-readonly memory (CD-ROM), a magneto-optical disk (MOD), a digital versatiledisc (DVD) and/or other suitable memory elements or combinationsthereof.

Any one or a combination of the flight management system and vehiclecontrol system may also include a communications interface. Thecommunications interface can include associated electronic circuitrythat is used to send and receive data. More specifically, thecommunications interface can be used to send and receive data betweenany of the various systems. Similarly, a communications interface at anyone of the systems may be used to communicate with outside componentssuch as another aerial vehicle and/or ground control. A communicationsinterface may be any combination of suitable wired or wirelesscommunications interfaces.

FIG. 5 is a flow diagram of a method 500 of machine vision processing ofsensor data of an aerial vehicle, and FIG. 6 is a flow diagram of amethod 600 of machine vision processing of sensor data of an aerialvehicle, according to example embodiments of the present subject matter.It should be understood that the operations of the methods disclosedherein are not necessarily presented in any particular order and thatperformance of some or all of the operations in an alternative order(s)is possible and is contemplated. The operations have been presented inthe demonstrated order for ease of description and illustration.Operations may be added, omitted, and/or performed simultaneously,without departing from the scope of the appended claims.

For example, the operations of the methods 500 and 600 are describedherein as being implemented, at least in part, by system components suchas controller 402, which can comprise a processor, control application,machine vision circuitry, machine vision components and/or machinevision applications. In some configurations, the system componentsinclude functionality produced by an application programming interface(API), a compiled program, an interpreted program, a network service, ascript, or any other executable set of instructions.

Although the following illustration refers to the components of FIG. 4,it can be appreciated that the operations of the methods 500 and 600 maybe also implemented in many other ways. For example, the methods 500 and600 may be implemented, at least in part, by a processor of a remotecontroller or a local circuit, including a remote pilot control centerfor controlling an unmanned aerial vehicle. In addition, one or more ofthe operations of the methods 500 and 600 may alternatively oradditionally be implemented, at least in part, by a chipset workingalone or in conjunction with software modules on board the aerialvehicle 200. Any service, circuit or application suitable for providingthe techniques disclosed herein can be used in operations describedherein, either remote to, onboard, or a combination thereof with respectto the aerial vehicle 200.

As shown in FIG. 5, the method 500 includes obtaining sensor data from asensor array 404 or 406, at block 502. The sensor data can include, forexample, optical data or radar data obtained from at least one sensor ofthe sensor arrays 404 and 406. Optical sensor data can also be obtained,and can include stereoscopic images, still images, or video recordedfrom an optical sensor of the sensor arrays 404 and 406. Other sensordata can include altitude, distance from ground, and other suitabledata.

The method 500 further includes obtaining performance data from avionics412 and/or sensor array 404 or 406, at block 504. The performance datacan include, for example, a calculated approach vector, a thrust vector,rate of ascent, rate of descent, angle relative to a ground plane, andother suitable performance data. In at least one example, theperformance data is obtained from sensors and/or avionics system 412.The performance data may be obtained directly from sensors and/oravionics system 412 or may be derived from sensor data obtained from thesensors. According to other examples, the performance data is inputmanually. Still according to other examples, the performance data may bebased on positions of control surfaces and engine status provided byavionics 412 or other components of the aerial vehicle 200.

The method 500 further includes processing the sensor data based on theperformance data, at block 506. In some examples, block 506 may includeprocessing the sensor data to compensate for the performance data. Insome examples, the sensor data is processed with a machine visioncontroller using machine vision and/or image processing techniques. Insome examples, the sensor data is processed to compensate for movementof the aerial vehicle as determined from the performance data. Accordingto at least one example, the processing can include processing adistorted image using machine vision to create an image with reduced,minimized, or corrected distortion. The corrected image may includelegible indicia and other corrections. The processing can also includeprocessing to compensate for movement, such as by, for example,correcting perspectives of buildings, landmarks, and other features.Additionally, the processing can include isolating landmarks, indicia,geographic features, and other optical data for further identificationin subsequent or parallel machine vision processing. For example, and asillustrated in FIG. 3B, the sensor data 310 can be processed to createcorrected sensor data 320. The machine vision controller 402 may processthe sensor data 310 to correct apparent distortion, misalignments, andother issues using the performance data and any other available data.

The method 500 further includes identifying a geographic indicator inthe processed sensor data, at block 508. For example, the machine visioncontroller 402 can interpret the processed sensor data to identifyvisual indicators such as those illustrated in FIG. 1. Other geographicindicators can include, for example, a number of buildings surroundingan landing area, height of buildings surrounding a landing area,surrounding higher elevation (such as mountains or natural features),surrounding lower elevation (such as crevasses or natural features), andother geographic indicators. Still further, geographic indicators caninclude GPS data, rivers, bodies of water, physical or virtual beacons,temporary structures such as towers/cones, and other indicators.

The method 500 further includes determining a geographic location of theaerial vehicle based on the identified geographic indicator, at block510. The machine vision controller 402 can compare the identifiedgeographic indicators to a predetermined set of geographic indicators,such as expected geographic features or a map or model of expectedfeatures, to determine a location of the aerial vehicle 200. Forexample, the machine vision controller 402 can determine if an expectedcontrol tower at an airport is detectable in the sensor data. Themachine vision controller 402 can also determine if airport buildingsare detectable in the sensor data. Thus, using all available sensordata, the machine vision controller 402 can compare the sensor data toexpected data from a flight plan to determine the geographic location.

In some example aspects, a model of expected features may include depthor height information. Accordingly, in some examples the machine visioncontroller 402 is configured to determine a height of geographicfeatures in a proximal location to the aerial vehicle based onprocessing the sensor data. For example, the machine vision controller402 can access one or more images obtained by one or more image sensorsto identify one or more geographic features. A depth camera,radar/LIDAR, and/or other sensors may be used to determinethree-dimensional or height information associated with the geographicfeatures. The machine vision controller 402 can access a database ofpredetermined geographic indicators associated with a geographiclocation such as known landing locations. The database can includeheight or other depth information associated with the predeterminedgeographic indicators. The machine vision controller 402 can compare thedatabase, including the height of geographic indicators, in order toidentify geographic features based on the image data. The machine visioncontroller 402 can determine a corresponding geographic location to ageographic feature based on comparing the height of geographic featuresfrom the sensor data to height information in the database.

Utilizing the determined geographic location, a pilot or operator of theaerial vehicle 200 can decide whether to safely land the aerial vehicle,alter a flight plan, or otherwise control the aerial vehicle. It iscontemplated that visual indicators such as warnings and audioindicators such as alarms can also be used. For example, an indicatorsuch as a visual indicator that landing can be performed safely or notsafely can be provided. In some implementations, the indicator mayidentify a runway or other feature with which the aerial vehicle isaligned for landing. In some implementations, an indicator may include awarning that the aerial vehicle is misaligned. For example, the systemmay compare the geographic indicator with flight plan data and provide awarning if the geographic indicator is not associated with the flightplan data. In some examples, a warning can be provided if geographicindicators include obstructions identified in machine vision processing.Furthermore, visual and audio indicators can include positiveindications that a landing pattern is deemed safe by the machine visioncontroller 402 such that the pilot or operator can concentrate on othermaneuvers to complete a safe landing.

Additionally, the pilot or operator of the aerial vehicle 200 may beremote, and the aerial vehicle may be an unmanned aerial vehicle, suchthat alerts, indicators, and geographic location are provided to thepilot or operator in a remote location. In these circumstances, theidentified geographic location may be used to automatically maneuver theaircraft, as described below with reference to FIG. 6. It is noted,however, that a manned aircraft may be automatically maneuvered inaccordance with embodiments of the disclosed technology. FIG. 6 is aflow diagram of an additional method 600 of machine vision processing ofsensor data of an aerial vehicle, according to example embodiments ofthe present disclosure.

As shown in FIG. 6, the method 600 includes obtaining sensor data from asensor array 404 or 406, at block 602. The sensor data may besubstantially similar to the sensor data described above. The method 600further includes processing the obtained sensor data to identifygeographic features in the area proximal to the aerial vehicle, at block604. The processing may include correcting distortion, compensating formovement, identifying obstacles, and/or matching flight plans toexpected geographic features.

According to at least one example, the processing can include processinga distorted image, or distorted sensor data, using machine vision tocreate a processed image or processed sensor data with reduced,minimized, or corrected distortion. The processed image may includelegible indicia and other corrections. The processing can also includeprocessing to compensate for movement, such as by, for example,correcting perspectives of buildings, landmarks, and other features.Additionally, the processing can include isolating landmarks, indicia,geographic features, and other optical data for further identificationin subsequent or parallel machine vision processing.

In some examples, processing the sensor data includes processing imagedata based on aircraft misalignment. For example, performance data suchas crosswind and other information can be used to determine amisalignment of an aircraft approach vector with the image data obtainedfrom one or more sensors. The performance data can be used to translateor otherwise determine modified sensor data to correct or compensate forthe misalignment. For example, when approaching a landing area the imagedata may identify a first runway according to a first runway marker forexample. However, the performance data may indicate that the aerialvehicle is landing in a strong crosswind. This and/or other performancedata may indicate that the aerial vehicle is misaligned with itsapproach vector. This information may be used to modify the sensor datato indicate the appropriate information corresponding with the approach.As a result, the machine vision controller 402 may determine that theaircraft is actually approaching a second runway adjacent to the firstrunway. The determination may be made even though the image datacorresponds to the first runway.

The method 600 further includes identifying a geographic indicator inthe processed sensor data, at block 606. The geographic indicator mayinclude a desired indicator or an obstacle. For example, the geographicindicator may include a landmark, body of water, airport, particularrunway at an airport, or other geographic indicator, including allgeographic indicators described herein. Using machine vision processingand/or a machine vision algorithm, the geographic indicator can beidentified in the processed sensor data. Additionally, the machinevision controller can identify obstacles at block 606. For example, anobstacle can include any object or indicia that is not expected in alanding area. Obstacles can include temporary or permanent obstacles.Obstacles can further include indicia purposefully marked (e.g., such asan ‘X’ or ‘NOT SAFE FOR LANDING’) to allow pilots and operators toidentify that a landing area is not to be used for landings. In someexamples, the machine vision controller 402 can immediately indicate thepresence of an obstacle. In some examples, the machine vision controller402 can delay an alert if an obstacle is a moving obstacle, such as alanding assist vehicle, that will be clear of an active landing areaprior to landing by the aerial vehicle. Furthermore, the machine visioncontroller 402 can issues multiple forms of soft alerts and high alertsif moving obstacles are deemed hazardous or are not moving as expected,or are not following an expected pattern. Processing and identifyingobstacles can be aided through use of radar/LIDAR information, and othersuitable data available from sensors.

The method 600 further includes determining if the identified geographicindicator is in a flight plan associated with the aerial vehicle, atblock 608. For example, the machine vision controller 402 may processthe sensor data and compare the same to expected sensor data of theflight plan. The expected sensor data may include maps, models,renderings, and/or other suitable data for machine vision processing andcomparison to the processed sensor data. The expected sensor data caninclude, for example, a two-dimensional map having associated height ordimensional information. Machine vision processing can facilitatecomparisons between the processed sensor data and this expected sensordata to determine if a match exists or if the geographic indicatorexists within the flight plan.

The method 600 further includes automatically maneuvering the aerialvehicle, or providing visual indication, based on the determination, atblock 610. For example, the aerial vehicle, based on the determination,may be automatically maneuvered to avoid a landing or continue flying.Other automatic maneuvers including obstacle avoidance, fast touchdownand takeoff, and similar maneuvers based on geographical indicators arealso applicable. The machine vision controller 402 can also providevideo or audio indications, as described above.

It should be appreciated that the operational blocks of method 500 andmethod 600 may not exhaustively describe all aspects of aerial vehiclecontrol. These operational blocks may be a simplified operational flowchart describing only partial aspects of aerial vehicle control, andshould not be construed of illustrating all possible control scenarios.

FIG. 7 depicts a block diagram of an example computing system 1000 thatcan be used to implement methods and systems according to exampleembodiments of the present disclosure. Computing system 1000 may be usedto implement a machine vision controller 402 as described herein. Itwill be appreciated, however, that computing system 1000 is one exampleof a suitable computing system for implementing the machine visioncontroller 402 and other computing elements described herein.

As shown, the computing system 1000 can include one or more computingdevice(s) 1002. The one or more computing device(s) 1002 can include oneor more processor(s) 1004 and one or more memory device(s) 1006. The oneor more processor(s) 1004 can include any suitable processing device,such as a microprocessor, microcontroller, integrated circuit, logicdevice, or other suitable processing device. The one or more memorydevice(s) 1006 can include one or more computer-readable media,including, but not limited to, non-transitory computer-readable media,RAM, ROM, hard drives, flash drives, or other memory devices.

The one or more memory device(s) 1006 can store information accessibleby the one or more processor(s) 1004, including computer-readableinstructions 1008 that can be executed by the one or more processor(s)1004. The instructions 1008 can be any set of instructions that whenexecuted by the one or more processor(s) 1004, cause the one or moreprocessor(s) 1004 to perform operations. The instructions 1008 can besoftware written in any suitable programming language or can beimplemented in hardware. In some embodiments, the instructions 1008 canbe executed by the one or more processor(s) 1004 to cause the one ormore processor(s) 1004 to perform operations, such as the operations formachine vision processing of sensor data, identifying geographicfeatures and indicators, locating an aerial vehicle based on sensordata, and other operations such as those described above with referenceto methods 500 and 600, and/or any other operations or functions of theone or more computing device(s) 1002.

The memory device(s) 1006 can further store data 1010 that can beaccessed by the processors 1004. For example, the data 1010 can includesensor data such as engine parameters, model data, logic data, etc., asdescribed herein. The data 1010 can include one or more table(s),function(s), algorithm(s), model(s), equation(s), etc. according toexample embodiments of the present disclosure.

The one or more computing device(s) 1002 can also include acommunication interface 1012 used to communicate, for example, with theother components of system. The communication interface 1012 can includeany suitable components for interfacing with one or more network(s),including for example, transmitters, receivers, ports, controllers,antennas, or other suitable components.

The technology discussed herein makes reference to computer-basedsystems and actions taken by and information sent to and fromcomputer-based systems. One of ordinary skill in the art will recognizethat the inherent flexibility of computer-based systems allows for agreat variety of possible configurations, combinations, and divisions oftasks and functionality between and among components. For instance,processes discussed herein can be implemented using a single computingdevice or multiple computing devices working in combination. Databases,memory, instructions, and applications can be implemented on a singlesystem or distributed across multiple systems. Distributed componentscan operate sequentially or in parallel.

Although specific features of various embodiments may be shown in somedrawings and not in others, this is for convenience only. In accordancewith the principles of the present disclosure, any feature of a drawingmay be referenced and/or claimed in combination with any feature of anyother drawing.

This written description uses examples to disclose the claimed subjectmatter, including the best mode, and also to enable any person skilledin the art to practice the claimed subject matter, including making andusing any devices or systems and performing any incorporated methods.The patentable scope of the disclosed technology is defined by theclaims, and may include other examples that occur to those skilled inthe art. Such other examples are intended to be within the scope of theclaims if they include structural elements that do not differ from theliteral language of the claims, or if they include equivalent structuralelements with insubstantial differences from the literal languages ofthe claims.

What is claimed is:
 1. An aerial vehicle, comprising: a plurality ofsensors mounted thereon; an avionics system configured to operate atleast a portion of the aerial vehicle; and a machine vision controllerin operative communication with the avionics system and the plurality ofsensors, the machine vision controller configured to perform a method,the method comprising: obtaining sensor data from at least one sensor ofthe plurality of sensors, wherein the sensor data is associated with aproximal location of the aerial vehicle and the proximal location iswithin sensor range of the aerial vehicle; determining performance dataassociated with the aerial vehicle; processing the sensor data based onthe performance data to compensate for movement of the unmanned aerialvehicle; identifying at least one geographic indicator based onprocessing the sensor data, wherein identifying the at least onegeographical indicator comprises comparing the sensor data to apredetermined set of sensor data, identifying an obstacle associatedwith the sensor data based on the comparing, and triggering an alertindicating identification of the obstacle; determining whether theobstacle is moving; and determining a geographic location of the aerialvehicle based on the at least one geographic indicator, wherein when theobstacle is determined to be moving, delaying the triggering of thealert.
 2. The aerial vehicle of claim 1, wherein processing the sensordata comprises: determining a height of a plurality of geographicfeatures in the proximal location; comparing the plurality of geographicfeatures with a database identifying a plurality of predeterminedgeographic indicators associated with known landing locations, thedatabase including a height identifier for each of the plurality ofpredetermined geographic indicators; wherein determining the geographiclocation of the aerial vehicle comprises identifying the geographiclocation of the aerial vehicle based at least in part on the heightidentifier for the at least one geographic indicator and the height of acorresponding geographic feature.
 3. The aerial vehicle of claim 1,wherein the sensor data is based on still images or video recording byan image capture device located on the aerial vehicle.
 4. The aerialvehicle of claim 1, wherein the sensor data is based on remote sensingassociated with a laser, Light Detection and Ranging (LIDAR), or radar.5. The aerial vehicle of claim 1, wherein the processing of the sensordata comprises processing the sensor data to reduce distortion based onthe performance data associated with the aerial vehicle.
 6. The aerialvehicle of claim 5, wherein processing the sensor data to reducedistortion comprises: determining a crosswind value; estimating anapproach vector based on the crosswind value and the performance data;and translating the sensor data based on the approach vector.
 7. Theaerial vehicle of claim 1, wherein identifying the at least onegeographical indicator comprises: comparing the sensor data to a modelof the proximal area; and matching a first geographic indicatorassociated with the sensor data to at least one second geographicalindicator associated with the model of the proximal area.
 8. The aerialvehicle of claim 1, wherein identifying the at least one geographicalindicator comprises: comparing the sensor data to a predetermined set ofsensor data; and identifying an obstacle associated with the sensor databased on the comparing.
 9. The aerial vehicle of claim 1, wherein themethod further comprises: determining that the geographic location ofthe aerial vehicle is absent from a flight plan of the aerial vehicle;and instructing the avionics system to perform a corrective maneuver inresponse to the determining that the geographic location of the aerialvehicle is absent.
 10. The aerial vehicle of claim 1, wherein the methodfurther comprises: determining, based on the performance data, whetherthe aerial vehicle is attempting a landing; determining whether the atleast one geographic indicator is associated with an appropriate landingarea for the aerial vehicle; and in response to determining that theaerial vehicle is attempting a landing and that the at least onegeographic indicator is associated with an appropriate landing area,providing a visual indication that the geographic location correspondsto an appropriate landing area.
 11. The aerial vehicle of claim 10,wherein the method further comprises: in response to determining thatthe aerial vehicle is not attempting a landing and that the at least onegeographic indicator is associated with an appropriate landing area,determining that the visual indication should not be displayed.
 12. Theaerial vehicle of claim 1, wherein identifying the at least onegeographical indicator comprises: comparing the sensor data to apredetermined set of sensor data; identifying an obstacle associatedwith the sensor data based on the comparing; and triggering an alertindicating identification of the obstacle.
 13. The aerial vehicle ofclaim 12, further comprising: when the obstacle is determined to bemoving, the alert triggered is one of a soft alert and a high alertdepending on whether the obstacle is deemed hazardous, not moving asexpected, or not following an expected pattern.
 14. The aerial vehicleof claim 1, wherein the at least one geographic indicator is a buildingsurrounding a landing area.
 15. The aerial vehicle of claim 1, whereinthe at least one geographic indicator is as natural feature.
 16. Theaerial vehicle of claim 1, wherein the performance data further includesat least one of a rate of ascent and a rate of descent.
 17. A method oflocating an aerial vehicle relative to a proximal location, the methodcomprising: obtaining sensor data from one or more optical sensorsmounted on the aerial vehicle, wherein the sensor data is associatedwith the proximal location and the proximal location is within sensorrange of the aerial vehicle; determining performance data from one ormore additional sensors mounted on the aerial vehicle; determining,based on the performance data, whether the aerial vehicle is attemptinga landing; processing the sensor data based on the performance data tocompensate for movement of the unmanned aerial vehicle; identifying atleast one geographic indicator based on processing the sensor data;determining whether the at least one geographic indicator is associatedwith an appropriate landing area for the aerial vehicle; determining ageographic location of the aerial vehicle based on the at least onegeographic indicator; and in response to determining that the aerialvehicle is attempting a landing and that the at least one geographicindicator is associated with an appropriate landing area, providing avisual indication that the geographic location corresponds to anappropriate landing area.
 18. The method of claim 17, wherein the methodfurther comprises: in response to determining that the aerial vehicle isnot attempting a landing and that the at least one geographic indicatoris associated with an appropriate landing area, determining that thevisual indication should not be displayed.
 19. An aerial vehicle,comprising: a plurality of sensors mounted thereon; an avionics systemconfigured to operate at least a portion of the aerial vehicle; and amachine vision controller in operative communication with the avionicssystem and the plurality of sensors, the machine vision controllerconfigured to perform a method, the method comprising: obtaining sensordata from at least one sensor of the plurality of sensors, wherein thesensor data is associated with a proximal location of the aerial vehicleand the proximal location is within sensor range of the aerial vehicle;determining performance data associated with the aerial vehicle;processing the sensor data based on the performance data to compensatefor movement of the unmanned aerial vehicle, wherein the processingcomprises: determining a crosswind value; estimating an approach vectorbased on the crosswind value and the performance data; and translatingthe sensor data based on the approach vector; identifying at least onegeographic indicator based on processing the sensor data; anddetermining a geographic location of the aerial vehicle based on the atleast one geographic indicator.